feat(cli): submit keyless job feedback by Max17190 · Pull Request #221 · firecrawl/cli

Max17190 · 2026-09-11T20:14:45Z

Why

Let agents discover and optionally submit evidence for keyless Search, Scrape, and Parse using the API contract in #4616.

Summary

Extend the existing firecrawl feedback command with task, assessment, observation JSON/files, and Parse document class. Keyless submissions omit Authorization.
Preserve job references and invitations in JSON output and stderr, including failed jobs, while preserving normal content on stdout.
Keep a short optional reminder in command help and stderr. Bundled skills ask agents to submit concise keyless feedback on observed result quality or missing coverage when the host permits it, especially for wrong, incomplete, blocked, or failed results. Feedback is not a completion requirement, and declined or terminally rejected feedback is not retried. Optional retries of retryable errors respect server timing.
Authenticated Search feedback instructions keep their existing wording, scoped to authenticated callers.
Default keyless Search to web results while retaining authenticated defaults, explicit sources, and the Map feedback auth gate.
Preserve API validation details and retry timing for keyless rejections. Preserve authenticated feedback contracts and preferences. Keep the preference helper in the existing command module.

Release after the API is deployed and Docs #1414 is published.

Test Plan

Revision 4ce3448: 40 focused tests passed across feedback and client configuration, including credential-less custom API error cases. TypeScript and formatting passed; all GitHub CI checks passed on that revision.
Cubic confirmed the custom API compatibility finding was addressed and reported zero new issues at 4ce3448.
Head 37ece40 adds documentation-only clarification of successful-submission idempotency and optional retries. PR diff whitespace checks and formatting passed. All current-head CI checks passed. Cubic reported zero issues across five files in the latest revision, with confidence 5/5; no unresolved code-review findings remain. Its shadow auto-approval policy still calls for human review.
Earlier validation covered invitation preservation, failed jobs, Parse document class, authenticated behavior, and CLI packaging.
Live end-to-end verification remains blocked by the API staging feedback lookup failure.

After an eligible call, print the job reference and a one-line optional reminder pointing to `firecrawl feedback <endpoint> <jobId> --help` instead of the full server message and flag template. Remove the daily submission rule and the exact attempt rate from help, the README, and bundled skills, and drop the retired DAILY_LIMIT_REACHED code.

Resolve conflicts with Alexandria session feedback and the Alexandria-only search command: - Send Alexandria session fields for endpoint alexandria and keyless task, assessment, document class, and observations for job feedback. - Keep the keyless feedback help beside the updated search action. - Keep the Alexandria guidance and relocated completion criteria in the bundled skills alongside the keyless feedback contract. - Describe keyless feedback as requested in exchange for free keyless use in the stderr reminder, help, README, and skills.

Point to feedback when a result is wrong, incomplete, blocked, or an error, matching the API invitation and MCP wording, instead of framing keyless use as an exchange for feedback. Updates the reminder, command help, README and skills.

cubic-dev-ai

Review completed against the latest diff

_{Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic}

cubic-dev-ai

All reported issues were addressed across 18 files

_{Shadow auto-approve: would not auto-approve. Auto-approval blocked because this review re-detected 1 unresolved P0–P2 issue already reported by Cubic.

Re-trigger cubic}

Max17190 · 2026-10-01T16:50:00Z

@cubic-dev-ai review this PR in full at the current head.

Use the API contract in firecrawl/firecrawl#4616 as the source of truth. Keyless feedback is optional for Search, Scrape, and Parse. Map feedback remains authenticated. Keyless evidence requirements are checked before submission; authenticated fields and preferences are retained. Search JSON preserves references even for empty results. Successful Scrape invitations are in data.metadata; failed jobs carry top-level metadata and are handled by the shared keyless request path, including HTTP 200 responses with success: false.

Please recheck the complete diff and the prior findings against the current implementation.

cubic-dev-ai · 2026-10-01T16:50:11Z

@cubic-dev-ai review this PR in full at the current head.

Use the API contract in firecrawl/firecrawl#4616 as the source of truth. Keyless feedback is optional for Search, Scrape, and Parse. Map feedback remains authenticated. Keyless evidence requirements are checked before submission; authenticated fields and preferences are retained. Search JSON preserves references even for empty results. Successful Scrape invitations are in data.metadata; failed jobs carry top-level metadata and are handled by the shared keyless request path, including HTTP 200 responses with success: false.

Please recheck the complete diff and the prior findings against the current implementation.
...

@Max17190 I have started the AI code review. It will take a few minutes to complete.

Max17190 · 2026-10-01T16:58:05Z

@cubic-dev-ai review this PR in full at the current head.

Use the API contract in firecrawl/firecrawl#4616 as the source of truth. Keyless feedback is optional for Search, Scrape, and Parse. Map feedback remains authenticated. Keyless evidence requirements are checked before submission; authenticated fields and preferences are retained. Search JSON preserves references even for empty results. Successful Scrape invitations are in data.metadata; failed jobs carry top-level metadata and are handled by the shared keyless request path, including HTTP 200 responses with success: false.

The latest revision also displays keyless validation details and retry timing while preserving authenticated error output.

Please recheck the complete diff and the prior findings against the current implementation.

cubic-dev-ai · 2026-10-01T16:58:15Z

@cubic-dev-ai review this PR in full at the current head.

Use the API contract in firecrawl/firecrawl#4616 as the source of truth. Keyless feedback is optional for Search, Scrape, and Parse. Map feedback remains authenticated. Keyless evidence requirements are checked before submission; authenticated fields and preferences are retained. Search JSON preserves references even for empty results. Successful Scrape invitations are in data.metadata; failed jobs carry top-level metadata and are handled by the shared keyless request path, including HTTP 200 responses with success: false.

The latest revision also displays keyless validation details and retry timing while preserving authenticated error output.
...

@Max17190 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

1 issue found across 19 files

Confidence score: 5/5

In parse.test.ts, the keyless parse path never reads FIRECRAWL_NO_ENDPOINT_FEEDBACK or sends an opt-out header, so this test doesn't verify the env var's effect. Update it to assert behavior the keyless path actually supports.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/__tests__/commands/parse.test.ts">

<violation number="1" location="src/__tests__/commands/parse.test.ts:77">
P3: This test stubs `FIRECRAWL_NO_ENDPOINT_FEEDBACK` and asserts the request headers, but the keyless parse path never reads that env var and never sends an opt-out header, so the stub has no effect and the assertion is an exact duplicate of the "posts to /v2/parse without auth" test's header check. Either remove the test or make it assert the actual opt-out-relevant behavior of this PR: the keyless parse still reports the feedback invitation on stderr even when the opt-out env is set (as `feedback-invitation.test.ts` pins down).</violation>
</file>

_{Shadow auto-approve: would require human review. Adds keyless feedback submissions with new required fields and an auth bypass for feedback on search/scrape/parse; needs human review of the authorization and API contract changes.

Fix all with cubic | Re-trigger cubic}

… tests

Max17190 · 2026-10-01T17:15:13Z

@cubic-dev-ai review the latest changes at the current head.

Removed the redundant Parse opt-out test reported in the latest review and the equivalent shared-request header tests. Invitation behavior remains covered by the helper and command tests plus actual API staging flows. The feedback auth gate now uses the same case normalization as the existing endpoint parser; regression tests verify that Search, Scrape, and Parse case variants remain keyless while Map variants retain authentication.

cubic-dev-ai · 2026-10-01T17:15:27Z

@cubic-dev-ai review the latest changes at the current head.

Removed the redundant Parse opt-out test reported in the latest review and the equivalent shared-request header tests. Invitation behavior remains covered by the helper and command tests plus actual API staging flows. The feedback auth gate now uses the same case normalization as the existing endpoint parser; regression tests verify that Search, Scrape, and Parse case variants remain keyless while Map variants retain authentication.

@Max17190 A review is already in progress. Try again after it finishes.

cubic-dev-ai

0 issues found across 4 files (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would require human review. Adds keyless feedback submission for search/scrape/parse, a new required-field contract, and an auth-gate bypass for feedback; this is an authorization/data-handling change tied to an unreleased API contract.

Re-trigger cubic}

…edback

Max17190 · 2026-10-03T03:01:59Z

@cubic-dev-ai review the complete current PR diff after syncing with main. Use firecrawl/firecrawl#4616 as the source of truth for optional keyless feedback. Verify the 24-hour window, evidence requirements, preserved job references on success and failure, ownership, one submission per job, separate attempt limits, no keyless refunds or quota resets, and preservation of authenticated feedback. Recheck prior findings against the current code.

cubic-dev-ai · 2026-10-03T03:02:08Z

@cubic-dev-ai review the complete current PR diff after syncing with main. Use firecrawl/firecrawl#4616 as the source of truth for optional keyless feedback. Verify the 24-hour window, evidence requirements, preserved job references on success and failure, ownership, one submission per job, separate attempt limits, no keyless refunds or quota resets, and preservation of authenticated feedback. Recheck prior findings against the current code.

@Max17190 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

1 issue found across 18 files

Confidence score: 4/5

The README.md can be read as starting the 24-hour window when the reference arrives, but the API measures it from job creation. Clarify that submissions are due within 24 hours of job creation.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="README.md">

<violation number="1" location="README.md:476">
P2: This can imply a fresh 24-hour window after receiving the reference, but the API deadline is based on the job timestamp. State that submissions must be made within 24 hours of job creation.</violation>
</file>

_{Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic}

cubic-dev-ai

0 issues found across 4 files (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would not auto-approve. Auto-approval blocked by 1 unresolved P0–P2 issue from previous reviews.

Re-trigger cubic}

…ck guidance Keyless guidance now asks agents to submit concise feedback on observed result quality or missing coverage when the host permits it, especially for wrong, incomplete, blocked, or failed results. It states that feedback does not determine whether the task is complete and should not be retried after a decline or rejection. The search skill no longer lists sending feedback as a completion criterion. The authenticated search feedback instructions return to their existing wording, scoped to authenticated callers.

cubic-dev-ai

All reported issues were addressed across 3 files (changes from recent commits).

_{Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic}

Max17190 · 2026-10-08T15:41:17Z

Please review the complete current PR diff against the current API contract in firecrawl/firecrawl#4616. Include changes carried by merge commits, since an earlier merge commit received a passing check without an AI review. Verify optional feedback wording, the 24-hour window, required evidence, trusted caller identity, preserved success and failure references, independent attempt limits, no keyless refunds, and authenticated compatibility. Recheck prior findings against the current code. @cubic-dev-ai

cubic-dev-ai · 2026-10-08T15:41:28Z

Please review the complete current PR diff against the current API contract in firecrawl/firecrawl#4616. Include changes carried by merge commits, since an earlier merge commit received a passing check without an AI review. Verify optional feedback wording, the 24-hour window, required evidence, trusted caller identity, preserved success and failure references, independent attempt limits, no keyless refunds, and authenticated compatibility. Recheck prior findings against the current code. @cubic-dev-ai

@Max17190 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 18 files

_{Shadow auto-approve: would not auto-approve because issues were found.

View guided diff | Turn on auto-fix | Re-trigger cubic}

cubic-dev-ai

0 issues found across 7 files (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would require human review. Adds optional keyless feedback for Search/Scrape/Parse, skipping the feedback auth gate, changing keyless Search defaults, and adding invitation output to stderr/JSON; these user-facing auth and product changes need human sign-off.

View guided diff | Turn on auto-fix | Re-trigger cubic}

Max17190 · 2026-10-08T16:23:47Z

@cubic-dev-ai review this PR. Please review the complete current PR diff, including changes introduced in merge commits, and re-evaluate prior findings against the current head. Check correctness, optional feedback guidance, authenticated compatibility, and alignment with the API contract in firecrawl/firecrawl#4616.

Keyless Search skill guidance now consistently says Consider submitting and requires a returned feedback invitation. Please verify the previous metadata.jobId versus metadata.feedback finding is addressed and review the full implementation.

cubic-dev-ai · 2026-10-08T16:24:00Z

@cubic-dev-ai review this PR. Please review the complete current PR diff, including changes introduced in merge commits, and re-evaluate prior findings against the current head. Check correctness, optional feedback guidance, authenticated compatibility, and alignment with the API contract in firecrawl/firecrawl#4616.

Keyless Search skill guidance now consistently says Consider submitting and requires a returned feedback invitation. Please verify the previous metadata.jobId versus metadata.feedback finding is addressed and review the full implementation.

@Max17190 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 19 files

_{Shadow auto-approve: would not auto-approve because issues were found.

View guided diff | Turn on auto-fix | Re-trigger cubic}

cubic-dev-ai

0 issues found across 2 files (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would require human review. Adds keyless job-feedback submission to the CLI: auth gate bypass for search/scrape/parse feedback, web-only default keyless Search, stderr invitations, and docs/skill updates. Decisive factors: auth/opt-out policy changes and dependency on an unverified API contract.

View guided diff | Turn on auto-fix | Re-trigger cubic}

cubic-dev-ai

0 issues found across 5 files (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would require human review. Adds keyless job-feedback submission to the CLI: auth bypass for search/scrape/parse feedback, web-only default keyless Search, stderr invitations, and docs/skill updates. Decisive factors: auth/opt-out policy changes and reliance on an unverified API contract.

View guided diff | Turn on auto-fix | Re-trigger cubic}

feat(cli): submit keyless job feedback

cf5a9ac

Max17190 mentioned this pull request Sep 11, 2026

feat(api): accept structured keyless feedback firecrawl/firecrawl#4616

Open

3 tasks

Max17190 added 16 commits September 11, 2026 15:23

docs(cli): distinguish keyless and authenticated feedback

521389c

fix(cli): forward feedback invitation preferences

cf929f4

docs(cli): clarify shared keyless feedback daily limit

02baf67

fix(cli): preserve keyless feedback and silence opted-out guidance

2f3855c

fix(cli): retain keyless invitations and optional submissions

aec2df9

feat(cli): support structured keyless feedback categories

80a8808

docs(cli): clarify search feedback source attribution

fc5f679

docs(cli): align keyless feedback evidence guidance

b9a61bb

docs: clarify keyless feedback limits and failure reasons

49c4737

merge: synchronize keyless feedback with main

328a8b8

style(test): format CLI terms refusal test

7a0847d

merge: sync main for keyless feedback

9ea44ef

refactor(feedback): simplify and align optional keyless guidance

8f6ce95

Max17190 marked this pull request as ready for review October 1, 2026 16:08

fix(search): use web results by default for keyless callers

b0d54d5

cubic-dev-ai Bot reviewed Oct 1, 2026

View reviewed changes

fix(feedback): retain auth boundaries and handle keyless failures

4cb90b3

fix(feedback): expose keyless validation details and retry timing

66a980f

cubic-dev-ai Bot reviewed Oct 1, 2026

View reviewed changes

Comment thread src/__tests__/commands/parse.test.ts Outdated

fix(feedback): preserve keyless endpoint aliases and remove redundant…

ba84855

… tests

cubic-dev-ai Bot reviewed Oct 1, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into max/enable-keyless-fe…

15c817d

…edback

cubic-dev-ai Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread skills/firecrawl-search/SKILL.md Outdated

Comment thread README.md Outdated

Comment thread README.md

Comment thread src/__tests__/utils/feedback-invitation.test.ts Outdated

fix: clarify keyless feedback submission requirements

713628d

cubic-dev-ai Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread skills/firecrawl/SKILL.md Outdated

Comment thread skills/firecrawl/SKILL.md Outdated

fix(cli): clarify keyless feedback requirements

dcb67ba

cubic-dev-ai Bot reviewed Oct 8, 2026

View reviewed changes

Comment thread skills/firecrawl/SKILL.md Outdated

Comment thread src/commands/feedback.ts Outdated

Comment thread src/utils/client.ts Outdated

Comment thread skills/firecrawl-search/SKILL.md Outdated

Comment thread src/__tests__/commands/scrape.test.ts Outdated

fix(cli): preserve feedback preferences and failure invitations

db2eb7d

cubic-dev-ai Bot reviewed Oct 8, 2026

View reviewed changes

docs: align optional keyless search feedback guidance

f852a69

cubic-dev-ai Bot reviewed Oct 8, 2026

View reviewed changes

Comment thread src/commands/feedback.ts Outdated

Max17190 added 2 commits October 8, 2026 11:30

fix: preserve custom API feedback error behavior

4ce3448

docs: clarify optional keyless feedback retries

37ece40

cubic-dev-ai Bot reviewed Oct 8, 2026

View reviewed changes

Conversation

Max17190 commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

Summary

Test Plan

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Max17190 commented Oct 1, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 1, 2026

Uh oh!

Max17190 commented Oct 1, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 1, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Max17190 commented Oct 1, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 1, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Max17190 commented Oct 3, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 3, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Max17190 commented Oct 8, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 8, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Max17190 commented Sep 11, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading